ci(#5509): add harness-eval static analysis for agent configurations by Benkapner · Pull Request #5510 · fullsend-ai/fullsend

Benkapner · 2026-07-23T08:42:40Z

Summary

Adds a CI workflow that runs harness-eval static analysis on every PR touching agent configuration files.

Scans both repo root (CLAUDE.md, AGENTS.md, skills/, commands/) and scaffold (internal/scaffold/fullsend-repo/)
Uses the recommended preset (structural errors block, quality warnings are advisory)
Security gate is advisory (non-blocking) until the team validates signal quality
SARIF output enables inline annotations on the PR diff via GitHub Code Scanning
Summary comment posted on each PR with scan results

No LLM calls, runs in seconds, fully deterministic. Uses the harness-eval GitHub Action with pinned checkout action SHA per repo convention.

Related Issue

Closes #5509

Testing

Workflow file is valid YAML
Paths filter limits runs to agent config changes only
Actions checkout SHA-pinned per repo security convention
Will validate on this PR once CI runs

Checklist

PR title follows Conventional Commits
Commits are signed off (DCO)

github-actions · 2026-07-23T08:43:00Z

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

github-actions · 2026-07-23T08:43:01Z

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

qodo-code-review · 2026-07-23T08:43:44Z

PR Summary by Qodo

CI: add harness-eval checks for agent configuration changes

⚙️ Configuration changes ✨ Enhancement 🕐 Less than 10 minutes

AI Description

• Add a PR-scoped CI workflow to statically analyze agent configuration files.
• Run harness-eval on repo root and scaffold directories, emitting SARIF for code scanning.
• Post a PR comment with results; block only on structural lint errors.

Diagram

graph TD
  A["PR touches agent configs"] --> B["Workflow: harness-eval.yml"] --> C["Checkout repo"] --> D["harness-eval action"] --> E["Upload SARIF"] --> F["GitHub Code Scanning"]
  D --> G["PR summary comment"]

High-Level Assessment

The following are alternative approaches to this PR:

1. Pin harness-eval action to a tag/SHA

➕ Reduces supply-chain risk vs referencing @main
➕ Improves reproducibility of lint/security results over time
➖ Requires periodic maintenance to bump the pinned ref
➖ May slow adoption of upstream fixes unless updated

2. Run harness-eval via a pinned container/CLI step

➕ Full control over exact tool version and execution environment
➕ Less coupling to upstream action implementation changes
➖ More workflow YAML and setup complexity
➖ Need to manage installation/auth and SARIF wiring manually

3. Make security gate blocking after an evaluation period

➕ Prevents risky agent-config changes once signal quality is validated
➕ Aligns with “shift-left” security expectations for config inputs
➖ Potential for developer friction if false positives remain
➖ Requires agreement on severity thresholds and remediation workflow

Recommendation: The overall approach (path-filtered PR workflow + SARIF annotations + PR comment) is a good fit and should stay. The main improvement to consider is pinning the harness-eval action reference (currently @main) to a tag or commit SHA to match the repo’s apparent supply-chain hardening (checkout is already SHA-pinned). Keeping the security gate advisory is reasonable initially; revisit making it blocking once false-positive rates are understood.

Files changed (1) +35 / -0

Other (1) +35 / -0

harness-eval.ymlAdd PR workflow to run harness-eval with SARIF + PR commenting +35/-0

Add PR workflow to run harness-eval with SARIF + PR commenting

• Introduces a GitHub Actions workflow triggered only when agent configuration paths change. Runs harness-eval against both the repository root and the scaffold directory using the recommended preset, uploads SARIF for code scanning annotations, and posts a PR comment; lint blocks only on errors and the security gate remains non-blocking.

.github/workflows/harness-eval.yml

github-actions · 2026-07-23T08:45:08Z

Site preview

Preview: https://455ddefe-site.fullsend-ai.workers.dev

Commit: cb21805f634dc551af099244dfe79387c745eadc

qodo-code-review · 2026-07-23T08:46:54Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. ~~Unpinned action reference~~ ✓ Resolved 🐞 Bug ⛨ Security

Description

The workflow runs a third-party action from the moving @main ref, so upstream changes can silently
change CI behavior or introduce a supply-chain compromise. This also breaks the repo’s established
SHA-pinning convention for Actions.

Code

.github/workflows/harness-eval.yml[25]

+      - uses: redhat-community-ai-tools/harness-eval/.github/actions/harness-eval@main

Relevance

⭐⭐⭐ High
Team has accepted pinning mutable refs to immutable commits for CI supply-chain/reproducibility;
@main is riskier than tags.
PR-#1682
PR-#2069

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new workflow references the harness-eval action via a mutable branch ref, while other workflows
in this repo pin actions by commit SHA.
.github/workflows/harness-eval.yml[23-26]
.github/workflows/lint.yml[15-29]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The workflow uses `redhat-community-ai-tools/harness-eval/...@main`, which is a mutable reference and can change without any change in this repository.

### Issue Context
This repo pins GitHub Actions to commit SHAs (see existing workflows) to reduce supply-chain risk and keep CI deterministic.

### Fix Focus Areas
- .github/workflows/harness-eval.yml[23-26]

### Suggested fix
- Replace `@main` with a specific commit SHA (or a tag + SHA pin if that’s your convention).
- Add a version comment (as done for other actions) so updates are deliberate and reviewable.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. ~~Overbroad workflow token perms~~ ✓ Resolved 🐞 Bug ⛨ Security

Description

The workflow grants pull-requests: write and security-events: write to the job that runs an
external action, unnecessarily expanding what that action can do with GITHUB_TOKEN. This conflicts
with the repo’s existing pattern of keeping write-scoped permissions out of jobs that process PR
content.

Code

.github/workflows/harness-eval.yml[R14-26]

+permissions:
+  security-events: write
+  contents: read
+  pull-requests: write
+
+jobs:
+  lint-and-security:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - uses: redhat-community-ai-tools/harness-eval/.github/actions/harness-eval@main
+        with:

Relevance

⭐⭐⭐ High
Repo previously split PR-write perms into separate gate job to avoid untrusted checkout jobs having
write scope.
PR-#2106

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new workflow grants write permissions broadly, whereas existing CI workflows document and
implement a pattern of isolating pull-requests: write from jobs that handle PR content.
.github/workflows/harness-eval.yml[14-35]
.github/workflows/e2e.yml[66-109]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The workflow-level permissions include `pull-requests: write` and `security-events: write`, and the job runs a third-party action. Least-privilege suggests limiting write-scoped permissions to only the steps/jobs that truly need them.

### Issue Context
In this repo, other workflows explicitly isolate `pull-requests: write` away from jobs that operate on PR content (see e2e.yml comments and job permissions).

### Fix Focus Areas
- .github/workflows/harness-eval.yml[14-35]

### Suggested fix
- Move permissions from workflow-level to job-level.
- If PR comments are optional, drop `pull-requests: write` and set `comment: "false"`.
- If PR comments are required, split into two jobs:
 - **scan job**: `contents: read` + `security-events: write` only.
 - **comment job**: `pull-requests: write` only, and post a summary using a trusted in-repo step/script based on scan outputs/artifacts.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

3. ~~Runner image not pinned~~ ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

The workflow uses ubuntu-latest, which is a moving target and can change the execution environment
over time. Other workflows in this repo pin to ubuntu-24.04, so this introduces avoidable CI drift
and inconsistency.

Code

.github/workflows/harness-eval.yml[21]
+    runs-on: ubuntu-latest

Relevance

⭐⭐ Medium
Repo already uses ubuntu-latest in key workflows; no clear precedent pushing pinned runner images.
PR-#1215
PR-#2106

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Only this new workflow uses ubuntu-latest; existing CI uses a pinned Ubuntu version, reducing
environment drift.
.github/workflows/harness-eval.yml[19-22]
.github/workflows/lint.yml[13-18]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`runs-on: ubuntu-latest` can change over time (image upgrades), which can introduce unexpected CI behavior changes.

### Issue Context
Other workflows in this repo consistently use `ubuntu-24.04`.

### Fix Focus Areas
- .github/workflows/harness-eval.yml[20-22]

### Suggested fix
- Change `runs-on: ubuntu-latest` to `runs-on: ubuntu-24.04` for consistency and reproducibility.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

rh-hemartin · 2026-07-23T09:32:01Z

Agent files will disappear soon from this repository, is this still valid with that fact?

Benkapner · 2026-07-23T09:38:42Z

@rh-hemartin and this repo will not have any harness files at all? agents, skills, commands, claude md hooks? if not so its not relevant but maybe in the new/other repo they will be?

rh-hemartin · 2026-07-23T09:53:57Z

@rh-hemartin and this repo will not have any harness files at all? agents, skills, commands, claude md hooks? if not so its not relevant but maybe in the new/other repo they will be?

No, they were moved to github.com/fullsend-ai/agents. We will have skills, commands and claude.md hooks here, but for our development usage, not for agents.

Benkapner · 2026-07-23T09:58:28Z

the main thing this tool does is lint + security checks if the user has committed any change to one of those harness files so could be relevant both here and the fullsend-ai/agents. wont hurt to add it to the CI. also it connects to docs/problems/trustworthiness-evidence.md @rh-hemartin

rh-hemartin · 2026-07-23T10:07:01Z

Ah, for a moment I was confused as well regarding the concept of harness, for us is a specific thing. Then it could be useful here, yes.

You need to pass pre-commit to your files BTW.

rh-hemartin

Incorporate this as a step in the current lint.yaml workflow, as it contains all the CI checks. The file filter can be done within a step, or just run it against everything. Optionally, since this does not require LLM, you could add it as a pre-commit (however this is more involved than the current setup, so don't worry, maybe a followup).

Benkapner · 2026-07-23T12:02:04Z

moved harness-eval into lint.yml as a step in the test job instead of a separate workflow. uses the checkout v7.0.1 already pinned there. will look into pre-commit integration as a follow-up.

rh-hemartin · 2026-07-23T13:07:24Z

You need to fix all the commits to ci(#5509) or ci(lint) or something similar. You could also squash them into a single one if you would like.

Benkapner · 2026-07-23T13:32:19Z

squashed into a single commit with ci(#5509) prefix. should pass commit-lint now.

rh-hemartin · 2026-07-24T09:11:09Z

Hey! Would you able to fix the problems raised by the tool?

Add harness-eval as a step in the existing lint.yml test job. Runs 74 deterministic lint rules against agent configurations (skills, commands, AGENTS.md, CLAUDE.md) in both repo root and internal/scaffold/fullsend-repo. Pinned to harness-eval==6.1.0 for renovate tracking. Signed-off-by: Benjamin Kapner <bkapner@redhat.com>

Benkapner · 2026-07-24T18:18:30Z

they were false positives of broken references. i fixed all of them (generic solution) in my repo and released harness-eval v6.1.1 with better path resolution for broken-references (checks project root, skill scripts/ subdir, skips code blocks, skips bare filenames). bumped to ==6.1.1 in this PR. the step now passes with 0 errors on this repo.

the 56 warnings it reports are all real quality findings (orphan skills, unfinished content, stale python version, taint flow in merge-queue). happy to open a separate PR for those if you want, or leave them as advisory. they are not something that should block @rh-hemartin

while i write this i see that the CI keeps failng. i think its because 6.1.1 is not publish yet. we should just wait a few minutes are re try

Benkapner requested a review from a team as a code owner July 23, 2026 08:42

github-actions Bot deployed to site-preview July 23, 2026 08:45 View deployment

qodo-code-review Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread .github/workflows/harness-eval.yml Outdated

Comment thread .github/workflows/harness-eval.yml Outdated

Comment thread .github/workflows/harness-eval.yml Outdated

github-actions Bot deployed to site-preview July 23, 2026 09:18 View deployment

rh-hemartin reviewed Jul 23, 2026

View reviewed changes

Comment thread .github/workflows/harness-eval.yml Outdated

rh-hemartin requested changes Jul 23, 2026

View reviewed changes

github-actions Bot deployed to site-preview July 23, 2026 11:57 View deployment

rh-hemartin reviewed Jul 23, 2026

View reviewed changes

Comment thread .github/workflows/lint.yml Outdated

github-actions Bot deployed to site-preview July 23, 2026 12:03 View deployment

Benkapner force-pushed the ci/harness-eval-lint branch from 4d397b2 to 310f489 Compare July 23, 2026 13:31

github-actions Bot deployed to site-preview July 23, 2026 13:34 View deployment

Benkapner force-pushed the ci/harness-eval-lint branch from 310f489 to cb21805 Compare July 24, 2026 18:14

github-actions Bot deployed to site-preview July 24, 2026 18:16 View deployment

Uh oh!

Conversation

Benkapner commented Jul 23, 2026 • edited by rh-hemartin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Testing

Checklist

Uh oh!

github-actions Bot commented Jul 23, 2026

E2E tests did not run

Uh oh!

github-actions Bot commented Jul 23, 2026

E2E tests did not run

Uh oh!

qodo-code-review Bot commented Jul 23, 2026

PR Summary by Qodo

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Site preview

Uh oh!

qodo-code-review Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rh-hemartin commented Jul 23, 2026

Uh oh!

Benkapner commented Jul 23, 2026

Uh oh!

rh-hemartin commented Jul 23, 2026

Uh oh!

Benkapner commented Jul 23, 2026

Uh oh!

Uh oh!

rh-hemartin commented Jul 23, 2026

Uh oh!

rh-hemartin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Benkapner commented Jul 23, 2026

Uh oh!

rh-hemartin commented Jul 23, 2026

Uh oh!

Benkapner commented Jul 23, 2026

Uh oh!

rh-hemartin commented Jul 24, 2026

Uh oh!

Benkapner commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Benkapner commented Jul 23, 2026 •

edited by rh-hemartin

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

qodo-code-review Bot commented Jul 23, 2026 •

edited

Loading